White-Box Testing refers generally to a tester (e.g., the person writing/designing the test case, who sometimes is a developer) who has special knowledge about the implementation details of the test subject being able to leverage this knowledge to test the subject thoroughly. Compared to traditional black-box testing, where the tester only comes in contact with visible parts of the test subject, white-box testing sometimes enables the tester to test parts of the subject that may not be visible from the outside. Both white-box and black-box testing test visible and hidden parts of the software, but a difference is that the knowledge of the test subject internals may lead to better tests in white-box testing. Black-box testing may be more meaningful when used in integration tests, whereas white-box testing may be suitable for unit testing involving small, separate possible internal parts of the subject.
Current GUI testing tools and frameworks frequently rely on capturing user input (e.g., in form of mouse actions and/or keyboard input), optionally translating them into human readable or understandable actions, and replaying them at a later time, in order to execute the test case.
In contrast, white-box testing typically involves potentially widespread unit testing. In unit testing, modules or small component parts of a program can be separately and automatically tested. In this regard, tools such as JUnit, for example, are becoming an increasingly important part of software development.
The following table helps differentiate black-box and white-box testing techniques:
AspectBlack-Box TestingWhite-Box TestingApplicationOnly the functionality ofThe tester, usually aKnowledgethe application isdeveloper, generally needs torequired (e.g., featuresknow the internal logic of theas described in userapplication (e.g., its programmanual). These testscode) in order to perform unitare generally performedby testers.testing.ProgrammingGenerally not required,Generally required inalthough some testingconnection with currentframeworks offertechniques.automation supportvia some scripting orprogramming languages.ApplicabilityGenerally high level: InGenerally low level: White-Levelmost cases, the testsbox testing is mainlysimulate actual usage byapplicable on a low level aspersons operating thein unit testing, where eachapplication to test.component/program entity istested separately withknowledge of its internallogic/a detailedspecification of itsprogramming interface.
There are a few commercially available programs for GUI testing, but they generally treat the GUI as a black box. In this vein, typical test frameworks mainly simulate user interaction by scripts, have means to read the screen and react on some events, and then verify a current state with an expected state. While some such frameworks and tools provide means for scripting or automating the tests with custom or standard programming languages, they generally do not provide access to data and classes of the target application domain. Thus, it is difficult and sometimes even impossible to implement custom test logic in the generated GUI test cases. But for white-box testing, the ability to implement custom test logic in the generated GUI test cases is taken as a given: If a test framework does not provide access to the application domain, it is impossible to do white-box testing and therefore impossible to even write custom test logic.
The reliability of information extraction also sometimes is a concern with current solutions. Standard GUI testing uses methods to read data from screen using generic component introspection at runtime, sometimes referred to as data scraping. These methods generally are thought to be more reliable than optical character recognition (OCR) because they deliver the exact character read from the screen, but they unfortunately leave room for interpretation errors (e.g., date parsing, Y2K problems, etc.). In some cases, displayed data may be truncated or transformed and therefore cannot be reliably used in tests. Although the read information may be available in its original and reliable form somewhere within the application model, it may nonetheless remain inaccessible for data scraping methods, thereby rendering these methods useless at least for cross checking results and so on.
Test cases in GUI testing usually include recorded interactions with the GUI and some checks to verify the test result(s). Because most of the test case logic includes recorded interactions, “generifying” these—which may involve building similar test cases that only differ in a few details (like checking a phone number instead of an email address)—may unfortunately require the user to record the interaction again and again for every detailed variation. Some frameworks offer the recorded interaction as a script that can be copied and modified, but it has been found that even these scripting capabilities still do not reach the flexibility and power of unit testing written in standard programming languages.
A main part of test cases generated with standard GUI testing tools includes data and information extracted from a program version that has been run with the testing tool. The information flow is unidirectional, from a running specific version of the program into a test case. Unfortunately, current tools do not provide a means for synchronizing this information, e.g., such as updating the test cases whenever relevant changes have been made to the program, or target of the test cases. This behavior reveals the lack of maintainability of such generated test cases. As a result, unit testing provides an advantage over black-box testing. Moreover, current GUI test cases usually use a proprietary data format or scripting language that, by nature, are not as widespread as standard programming languages like Java. This can also lead to poor maintainability, e.g., given the special knowledge needed in dealing with such proprietary features. In contrast, unit test cases can coexist with the program's code base and can be maintained side-by-side as the program code evolves during time, potentially in connection with a more widespread (and thus potentially more maintainable) programming language.
When a standard GUI testing tool is used, the test coverage is difficult or even impossible to assess automatically. There typically is no support for test coverage means of current Integrated Development Environment (IDEs). As an application evolves over time, assessing test coverage can become even more important (e.g., so that nothing is left untested) but also can be very difficult to assess without the help of dedicated tools (which are not supported in case of current GUI testing technologies). On the other hand, unit testing uses code coverage metrics efficiently because of the integration with the application code base. Moreover, in case of unit tests, it is even possible to assess the quality of the tests themselves using, for example, mutation testing. By contrast, it is highly improbable that such advanced techniques can be carried on normal GUI test cases.
Standard unit testing currently cannot be directly applied on GUI testing without the support of adapters and specialized frameworks. The more or less current standard unit testing tool, JUnit, is not appropriate for test cases dealing with GUI components. This is caused by two problems, namely, problems associated with building test cases and running them. With respect to the former, GUIs and GUI component oftentimes are more interesting to test in a way similar to integration tests, in that it oftentimes is desirable to test many components together in terms of interaction and as whole block. Unfortunately, it is very difficult to put components together so that they match the same constellation used in the application to test. With respect to running test cases, GUIs oftentimes run on a special thread dedicated to the user interface. Unit tests can be run on arbitrary threads but generally need to be adapted to run on the GUI thread. In case of Java and JUnit, for example, Java GUIs generally must be run on a special GUI thread, called the Event Dispatching Thread (EDT); otherwise, the results are undefined. JUnit does not currently provide any means to run test cases on the EDT. Therefore, a special test runner is needed, although such a tool is not yet commercially available.
Unit testing typically uses so called mock objects to simulate the behavior of the system parts that lie outside the scope of a test case. The mock objects may range from simulating the interaction with a simple class, to the communication with an external service like a database or the Internet. Mocking reduces the dependency on external systems and increases the reproducibility of certain cases subjected to tests. Both of these aspects are desirable in any kind of testing. However, in the case of standard GUI testing, the standard/established test framework can neither control the outside world, nor provide means to mock it. This makes standard GUI test cases something more akin to integration testing, mainly because of the dependence on the behavior of systems or program part not targeted by the test case.
Thus, it will be appreciated that there is a need in the art for white-box GUI testing techniques. For instance, it will be appreciated that it would be desirable to provide techniques that would, among other things, enable a developer to write unit tests for GUI parts and test them programmatically and extensively.
One aspect of certain example embodiments relates to supporting white-box GUI testing within a unit testing approach.
Another aspect of certain example embodiments relates to the introduction of an Event Dispatching Thread (EDT) adapter, which aids in the synchronization of GUI testing inside unit tests.
Another aspect of certain example embodiments relates to the inclusion of an interaction recorder in the application under test, which advantageously allows for the application to be run until a desired point, at which time custom test logic can be introduced.
Still another aspect of certain example embodiments relates to the integration of GUI testing into unit testing, advantageously allowing the IDE to produce statistics relevant to the use and correct functioning of GUI classes.
An advantage of certain example embodiments relates to the ability to provide automations for environment simulation and assertion sets.
Clear and well identified separation between the regular application and application under test is achieved using executable code injection in certain example embodiments.
In certain example embodiments, a method of testing an application in connection with a predefined test case is provided. The test case to be run on the application is retrieved. The test case includes (a) interactions between the application and components external to the test case's scope that were gathered as the application ran and received user input from a user until a desired test state was reached, and (b) assertions to be made to the application once the application reaches the desired test state. A test environment is set up in connection with a test runner. The test environment replaces a normal environment in which the application runs, and it includes an adapter configured to execute interactions on the test environment's GUI thread. The application is fed with interactions included in the test case, through the adapter, until the application reaches the desired test state. Once the application reaches the desired state, the application is fed with assertions included in the test case through the adapter. Expected data associated with the assertions is compared with data actually being generated by the application. It is determined, from the comparison, whether the test case resulted in a success or failure.
In certain example embodiments, a method of generating a test case for an application to be tested is provided. The application to be tested is started in an environment in which the application is normally run. Interactions between the application and components external to the test case's scope are stored, in connection with a recorder attached to the application, as the application runs and receives user input from a user. When a desired test state is reached: a determination is made as to which components of the environment are currently active; a determination is made, from a store including a plurality of different assertion sets, as to which of those assertions sets apply to the currently active components; corresponding component state data is fed to each said applicable assertion set to generate a corresponding assertion that holds current component state data; and the test case is generated, with the test case including the generated assertions and the stored interactions.
In certain example embodiments, a computer system for testing a computer application in connection with a predefined test case is provided. Processing resources include at least one processor and a memory. A non-transitory computer readable storage medium stores a plurality of test cases, with each said test case including (a) interactions between the application and components external to the test case's scope that were gathered as the application ran and received user input from a user until a desired test state was reached, and (b) assertions to be made to the application once the application reaches the desired test state. A test runner is configured to cooperate with the processing resources to set up a test environment in connection that replaces a normal environment in which the application runs. The test environment includes an adapter configured to execute interactions on the test environment's GUI thread. In connection with a selected one of said test cases, the adapter and/or test runner is/are configured to: feed the application with interactions included in the selected test case until the application reaches the desired test state; feed the application with assertions included in the selected test case once the application reaches the desired state; compare expected data associated with the assertions with data actually being generated by the application; and determine, from the comparison, whether the selected test case resulted in a success or failure.
In certain example embodiments, there is provided a non-transitory computer readable storage medium tangibly storing instructions that, when executed by at least one processor of a system, perform a method as described herein.
These aspects, features, and example embodiments may be used separately and/or applied in various combinations to achieve yet further embodiments of this invention.